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Abstract: An accurate real-time operator functional state assessment makes it possible to perform task 
management, minimize risks, and improve mission performance. In this paper, we discuss the 
development of an individualized operator functional state assessment model that identifies states likely 
leading to operational errors. To address large individual variations, we use two different approaches to 
build a model for each individual using its data as well as data from subjects with similar responses. If a 
subject’s response is similar to that of the individual of interest in a specific functional state, all the training 
data from this subject will be used to build the individual model. The individualization methods have been 
successfully verified and validated with a driving test data set provided by University of Iowa. With the 
individualized models, the mean squared error can be significantly decreased (by around 20%). 


1. INTRODUCTION 

In recent years, researchers have been actively 
performing machinery/electronics diagnostics and 
prognostics for automated aviation systems. To 
ensure mission success, the functional states of 
human operators also need to be monitored since 
mismatched Operator Functional State (OFS) and 
workload (either over-load or under-load) 
conditions can lead to disastrous consequences 
[ 1 ]. 

According to [2], OFS can be defined as the 
multidimensional pattern of human psycho 
physiological condition that mediates performance 
in relation to physiological and psychological 
costs. Different contributing factors, including 
environmental factors (Altitude, noise, etc.), 
individual state (circadian rhythms, sleep loss, 
illness, etc.), and task characteristics (physical 
load and cognitive load), can affect the OFS and 
lead to suboptimal performance in human 

operators. It is challenging to consider all those 
factors to predict the OFS accurately in real time. 
Furthermore, current available OFS modeling 
tools have limited applicability as they do not 
account for the considerable individual differences 
due to individual physical fitness and adaptability 
to external/internal conditions. 


In this paper, we introduce a closed-loop Adaptive 
Task Management System (ATMS) to identify 
hazardous states that are likely to lead to 
operational errors and dynamically aid operators 
to minimize human errors. Key innovations in the 
framework include 1) a systematic approach to 
perform OFS assessment considering all the 
contributing factors, 2) a committee machine- 
based regression model with advanced feature 
selection method to accurately build the mapping 
between input parameters and output functional 
state, 3) a two-step model individualization 
technique for individual OFS monitoring, and 4) 
efficient task management to address both over- 
load and under-load situations. An accurate OFS 
assessment is the foundation of the ATMS. 
Therefore, in this paper, we focus on how to build 
the real-time individual OFS assessment model. 

This paper is organized as follows. In Section 2 
we describe the closed-loop ATMS framework. In 
Section 3, we introduce the enhanced committee 
machine-based OFS assessment method. The 
focus on section 4 is to build the individualized 
OFS assessment model. In Section 5, the 
individualized OFS assessment model 
performance is verified and validated with a 
driving test dataset. Section 6 concludes this 
paper. 
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2. Closed-Loop ATMS Framework 



Figure 1: ATMS framework 


The framework of the closed-loop Adaptive Task 
Management System (ATMS) is shown in Figure 
1. In the ATMS framework, to accurately monitor 
the Operator Functional State (OFS) for each 
individual, the OFS assessment model is trained 
and individualized using different sources of 
training inputs (physiological signals, system 
dynamics measurements, etc.). An enhanced 
committee machine-based OFS assessment 
model is employed to map input parameters to 
individual OFS, in which the responses of multiple 
neural networks (committee members) are 
combined into a single response to improve 
efficiency and accuracy. To further boost the OFS 
assessment performance, we utilize an advanced 
feature selection algorithm [3] to select different 
features for each committee member. 

Due to large individual variations, a generalized 
OFS assessment model trained using data from 
large number of subjects usually does not yield 
satisfactory performance when applied to an 
individual operator. We individualize the 
generalized OFS assessment model using data 
from the individual of interest, as well as selected 
subjects whose data has been used in training the 
generalized model. The selection is based on a 
similarity measure: if a subject’s training data is 
similar to the individual’s data in specific functional 
states (for example, they are close to each other 
in the sense of Euclidian distance in the feature 
space computed from the data), all the training 


data from this subject will be used to individualize 
the generalized model. 

If the OFS of an individual decreases below a 
certain threshold, a task performance 
augmentation strategy can be applied to even-out 
workload and maintain the operator in an optimum 
cognitive workload level. As a result, the operator 
can be continuously engaged and able to respond 
quickly and appropriately to unusual situations. 

3. Enhanced Committee Machine-based OFS 
Assessment 

The basic procedure for real-time OFS 
assessment is shown in Figure 2. It includes pre- 
processing, feature extraction/selection, and 
regression. We have developed an enhanced 
committee machine-based regression method for 
the OFS assessment. 


EEG 

ECG 

Temperature 
Heart Rate 
Heart Rate 
Variability 
Ground Speed 



Figure 2: Real-time OFS assessment procedure 
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3.1 Committee Machine 

A committee machine is a strategy to improve 
classification or regression performance by 
combining responses from multiple diversified 
committee members (trained perceptrons in 
neural networks, for example) The performance of 
the committee machine is often better than that of 
each committee member [4] based on two main 
reasons. First, if committee members have the 
diversity property, i.e. they are unlikely to make 
errors in the same feature space, the errors from 
individual committee members will be canceled by 
each other to some extent. Second, since the 
committee machine “averages” its individual 
member’s estimation, the variance of the 
committee machine can be significantly reduced. 

Two types of committee machines, as shown in 
Figure 3, are implemented using a multilayer 
perceptron trained by the standard Back 
Propagation (BP) algorithm as the base 
regression model for OFS assessment. The base 
regression model is combined with an adaptive 
learning factor to make training algorithms 
converge much faster than the traditional BP [5]. 
Also, both committee machines are combined with 
an advanced feature selection algorithm, 
Piecewise Linear Orthogonal Floating Search 
(PLOFS) [3]. 



Figure 3: Two types of committee machines 


The first type of committee machine was built by 
training each committee member using different 
initial weights. It is trained based on an 
assumption that each member will converge to a 
different location in the solution space. Given the 
fact that the error surface of a neural network has 
multiple local minima, the local minimum that a 
training algorithm converges to will differ 
depending on its initial condition. Therefore, we 
combine the predictions from different committee 
members as the final solution. Each member with 
the diversity property covers partially the solution 


space, and the final committee can then cover 
larger solution space. 

Type 2 committee machine contains a set of 
multilayer perceptrons trained by different 
bootstrapped datasets after feature selection. To 
make each of the committee member diversified, 
each member further uses a different set of 
features, which is different from the traditional 
“bagging” training technique that simply 
aggregates bootstrapped individuals and selects 
the same features for all the committee members. 

3.2 Feature Extraction and Selection 

Many features can be extracted for OFS 
assessment. For EEG, total spectral power can be 
calculated in the alpha, theta, beta, and gamma 
bands as these bands reflect cognitive states. In 
addition, we can examine signal coherence 
between inter-hemispheric electrodes such as FI 
and F2 (from the 10-20 electrode placement 
system). Measures of signal coherence focus on 
the high-alpha bands (9-12 Hz), as increased 
coherence among these signals is thought to 
distinguish higher levels of cognitive activity. With 
the eye tracking data, we can examine blink 
frequency, percent eye closure (PERCLOS), 
average eye closure speed (AECS), 
mean/variation change of pupil size over time and 
the percentiles of pupil size. We can also extract 
features related to eye movements. The 
increased frequency of saccades may indicate an 
increase in multi-tasking demands, requiring 
operators to split attention. It may also indicate 
increased demands on spatial working memory, 
as operators may need to maintain visual data to 
integrate it across multiple displays. 

With the large amounts of features, we need to 
carefully evaluate the features and select a subset 
of features that can best estimate the OFS. A 
feature selection algorithm usually evaluates the 
fitness of features first, and then searches for 
different combinations of features with the goal of 
maximizing the fitness value [6-11]. Two common 
types of features selection algorithms are filter 
approaches and wrapper approaches. A filter type 
method ranks features according to some 
predefined criteria such as mutual information, 
class separability measure without any actual 
model assumed between outputs and inputs of 
the data, a feature is then selected or discarded 
based upon the ranking. A wrapper approach 
utilizes a model to evaluate the fitness values of 
features and features are selected using the 
fitness as a guide. Usually, wrapper approaches 
give better results than filter approaches but have 
higher computational complexities [7]. In the OFS 
assessment model, we utilize a wrapper type 
algorithm, Piecewise Linear Orthogonal Floating 
Search (PLOFS), to select features for the 

3 


225 















committee members [8]. What is unique of PLOFS 
is that its computational speed is similar to a filter 
approach. 

The PLOFS algorithm accumulates all necessary 
information in the auto- and cross- correlation 
matrices that are needed for feature selection in 
just one data pass. The feature searching 
procedure is then performed by evaluating 
goodness of a piecewise linear network through 
the auto- and cross- correlation matrices without 
passing through the original dataset. This is 
possible because of the orthogonal least square 
procedure, which makes the algorithm extremely 
efficient compared with other wrapper type 
algorithms. Other advantages of the PLOFS 
algorithm are as follows: 1) it selects features 
rather than a combination of all the available 
features such as those selected by transformation 
based methods (e.g., PCA, Wavelet); 2) it 
considers interactions among features and 
measures the correlations via the amount of 
explained variance by features; and 3) the 
algorithm produces a list of best combinations that 
contain different numbers of features, users then 
have the flexibility to choose any set based on 
their preferred criterion. 

4. Model Individualization 

Current available OFS modeling tools have limited 
applicability due to the fact that they do not 
account for the considerable individual differences 
in response to task schedule, individual fitness 
(sleep loss, anxious), and environmental changes. 
To improve the OFS estimation performance, 
these individual differences should be considered 
while building the OFS model. 

A straightforward approach to building an 
individual model is to utilize all the available data 
from the individual. This approach can achieve the 
best performance if the training information is 
sufficient to cover all the individual’s functional 
states. However, in many cases, training data for 
an individual is limited and is expensive to collect. 
Therefore, it may be infeasible to train such an 
individual OFS model. In this paper, we introduce 
two different approaches to address the limitation. 

The first individualization approach, Individual 
Model 1, is shown in Figure 4. 

In a specific functional state, one or more subjects 
may have similar responses. Therefore, to train an 
OFS model for an individual, we use the data from 
the individual, together with data from some 
similar subject(s), whose responses are similar to 
the individual. The similarity is measured by a 
metric computed based on the input features. 
Candidate methods to derive such a similarity 
measure include Euclidian distance, Principal 


Component Analysis and Mahalanobis distances 

[ 8 ]. 
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Figure 4: Individual Model 1 


We can find similar subjects in each functional 
state that the individual has experienced. After 
scanning all the functional states of the individual, 
we can select a set of subjects based on the 
similarity metrics. All the data from these subjects 
are then extracted as the training data for the OFS 
modeling of the individual. 

Another approach to address the limited training 
data issue, Individual Model 2, is based on the 
tuning of a generalized model. Using the extended 
data set as described above, the basic idea of 
Individual Model 2 is to select committee 
members that are sensitive to the individual’s 
OFS, and then tune the fusion weights for each of 
these committee members, i.e., perceptrons 
trained by the standard BP algorithm. This method 
is shown in Figure 5. 



Figure 5: Individual model 2 

More specifically, in the generalized model, we 
assume that all committee members are sensitive 
to the OFS of each individual and use a simple 
average to combine the output from each 
committee member. To individualize this 

generalized model, we will follow two steps: use 
the PLOFS algorithm to select a subset of 
committee members that are sensitive to the 
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individual’s OFS, and then adjust the weights of 
each of the selected committee members using a 
linear regression approach to form a final 
estimation of the OFS. Figure 6 compares the 
individualization process (right) to a generalized 
OFS assessment model (left). 



Figure 6: Model individualization 

5. Experimental Study 

We utilized the driving test dataset collected using 
the Cognitive Avionics Tool Set (CATS) software 
developed by Operator Performance Laboratory 
(OPL) [12] in the University of Iowa to verify and 
validate the individualized OFS assessment 
approach. The enhanced committee machine and 
individualization strategies were implemented on 
the driving test dataset. 

CATS [12] is a powerful, database driven data 
visualization and analysis package. The toolset 
synchronizes a large number of incoming data 
streams operating at different update rates into a 
single, unified file. CATS provides a rich set of 
visualization tools to inspect physiological data. 
Figure 7 shows a snapshot of the CATS software. 



Figure 7 CATS GUI 


In the driving test dataset, subjects performed a 
driving test in two hours. During the test, different 
types of information were collected, including 
description of the task, system dynamics related 


information (such as ground speed), performance 
measures, physiological signals (128-channel 
EEG, ECG, respiration, etc.), and eye tracking 
data. And the workload was analyzed every 
second based on the driving scenario (city-driving, 
stopped, highway passing, etc.). 

Five subjects’ data were used to verify and 
validate the individualized OFS assessment 
methods. We first trained a generalized model for 
each subject using a leave-one-out method, which 
trains the model using the other four subjects’ 
data and leaves the data from the subject being 
modeled for testing (Method 1 in Figure 8). We 
then implemented the previously described model 
individualization methods for each subject 
(Method 2: Individual Model 1; and Method 3: 
Individual Model 2). For a comparison purpose, 
we also trained an individual model for each 
subject by randomly selecting half of the data from 
that subject, and tested the model with the rest 
half (Method 4). The performance, Mean Squared 
Error (MSE), is compared in Figure 8 and Table 1. 



Figure 8: Performance comparison 


It is clear that the two model individualization 
methods (Method 2 and Method 3) significantly 
reduce the MSE given by generalized models 
(Method 1). Compared with a generalized OFS 
assessment model, the mean squared prediction 
error is about 20% lower. The performance using 
the two model individualization techniques are 
comparable to that of the individual model trained 
with sufficient data from the individual, but only 
require very limited data for training or 
individualization (5-minute in our experiment). 
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Table 1: Performance comparison 


Method 

Subject 1 
MSE ± STD 

Subject 2 
MSE ± STD 

Subject 3 
MSE ± STD 

Subject 4 
MSE ± STD 

Subject 5 
MSE ± STD 

1 

1.25 ±0.029 

1.18 ±0.024 

1.22 ±0.023 

1.01 ±0.021 

0.98 ±0.02 

2 

0.82 ±0.018 

0.76 ±0.018 

1.03 ±0.021 

0.79 ±0.017 

0.72 ±0.018 

3 

1.12 ±0.02 

0.67 ± 0.02 

1.16 ±0.026 

0.85 ±0.016 

0.87 ±0.022 

4 

1.01+0.009 

0.81 ±0.016 

0.67 ±0.017 

0.56 ±0.017 

0.7510.015 


6. Conclusions 

The research effort results in a successful 
development of an individual OFS assessment 
model for closed-loop task management. The 
model incorporates novel committee machine- 
based OFS assessment with an advanced feature 
selection method, Piecewise Linear Orthogonal 
Floating Search (PLOFS), and two different 
individualization techniques have been developed 
to improve the OFS assessment performance for 
each individual. The experimental results show 
significant improvements of the individualization 
techniques. 
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